A Knowledge-Based Approach to Understanding Students’ Explanations

High-precision Natural Language Understanding is needed in Geometry Tutoring to accurately determine the semantic content of students’ explanations. The paper presents an NLU system developed in the context of the Geometry Cognitive Tutor. The system combines unification-based syntactic processing with description logics based semantics to achieve the necessary accuracy level. The paper describes the compositional process of building the syntactic structure and the semantic interpretation of NLU explanations. It also discusses results of an evaluation of classification performance on data collected during a recent classroom study. 1. Explanations in Geometry Tutoring The Geometry Cognitive Tutor assists students in learning by doing as they work on geometry problems on the computer. Currently the Geometry Cognitive Tutor is in regular use (two days per week) in about 150 schools around the US. In previous evaluation studies Koedinger et al. [1] have shown that the tutors are successful in raising high school students’ test scores in both algebra and geometry. However, there is still a considerable gap between the effectiveness of current cognitive tutor programs and the best human tutors [2]. Cognitive Tutors pose problems to students and check their solutions to these problems step by step. They can also provide context-sensitive hints at each step in solving the problem, as needed. However, prior Cognitive Tutors do not ask students to explain or justify their answers in their words. On the other hand human tutors often engage students in thinking about the reasons behind the solution steps. Such “self-explanation” has the potential to improve students’ understanding of the domain, resulting in knowledge that generalizes better to new situations. This difference might also be the main explanation beneath the gap mentioned above. To verify this hypothesis, the next generation of intelligent cognitive tutors needs to be able to carry tutoring dialogs with students at the explanation level. Some of the current intelligent tutoring systems, like Autotutor [3], Circsim-Tutor [4], and Atlas/Andes [5], do have natural language processing capabilities. However, these systems rely on either statistical processing of language, identifying keywords in language, or some level of syntactic analysis. None of these approaches seem to achieve the degree of precision in understanding needed in a highly formalized domain such as geometry tutoring. One of the main problems that the Geometry Tutor faces is to determine with accuracy the semantic content of students’ utterances. Natural language allows for many different ways to express the same meaning, all of which have to be recognized by the system as being semantically equivalent. The determination of semantic equivalence has to work reliably over variation of syntactic structure, variation of content words, or a combination of both. For example, the sentences below all express the same geometry theorem, about the measures of angles formed by other angles (the Angle Addition Theorem). An angle formed by adjacent angles is equal to the sum of these angles. The measures of two adjacent angles sum up to the measure of the angle that the 2 angles form. An angle's measure is equal to the sum of the two adjacent angles that compose it. The sum of two adjacent angles equals the larger angle the two are forming. The sum of the measures of two adjacent angles is equal to the measure of the angle formed by the two angles. The measure of an angle made up of two adjacent angles is equal to the sum of the two angles. If adjacent angles form an angle, its measure is their sum. When an angle is formed by adjacent angles, its measure is equal to the sum of those angles. An angle is equal to the sum of its adjacent parts. Two adjacent angles, when added together, will be equal to the whole angle. The sum of the measures of adjacent angles equals the measure of the angle formed by them. Two adjacent angles added together make the measure of the larger angle. The process also has to be consistent, so no unwarranted conclusions are derived from the text, and robust, in an environment of imprecise or ungrammatical language, as uttered more often than not by high school students. Many times this content equivalence relies on inferences specific to the domain of discourse. Our hypothesis is that such a high-precision recognition process needs to be based on contextual information about the domain of discourse modeled in a logic system. The paper presents an NLU system we have built to test this hypothesis. The next section describes the overall architecture of the system, and illustrates the main interpretation mechanism. Section 3 discusses the results of an evaluation we completed based on data from a recent classroom study. 2. The System’s Architecture The system’s overall architecture is presented in Figure 1 below. The interface module takes the input sentence from the tutor, word by word, in real time, and after some preprocessing and spelling correction, it passes it to a chart parser. The chart parser is the main engine of the system. It uses linguistic knowledge about the target natural language from the unification grammar and the lexicon. The parser used currently is LCFlex, a left-corner active-chart parser developed at Carnegie Mellon University and University of Pittsburgh [6]. The parser takes words of a sentence one by one and combines them in larger phrase structures, according to rules in the unification grammar. It then calls the feature structure unifier in order to process restrictions attached to grammar rules and build feature structures (FS) for each phrase successfully recognized. These feature structures store lexical, syntactic, and semantic properties of corresponding words and phrases. The parser uses an active chart that serves as a storage area for all valid phrases that could be built from the word sequence it received up to each point in the process. Some of the restrictions in the grammar are directives to the description logics system, currently Loom [7]. The logic system relies on a model of the domain of discourse, encoded as concepts, relations, and production rules, in the two knowledge bases. Concepts and relations stand for predicates in the underlying logic. Production rules perform additional inferences that are harder to encode into concepts and/or relations. The linguistic inference module mediates the interaction between the feature structure unifier and the description logics system. This module is responsible for performing semantic processing that is specific to natural language understanding, like compositional semantics, resolving metonymies and references, and performing semantic repairs. Based on this knowledge base, the logic system builds compositionally a modeltheoretic semantic representation for the sentence, as a set of instances of various concepts connected through various relations. An instance corresponds to a discourse referent in the sentence. The logic system performs forward-chaining classification of resulting instances, and also ensures semantic coherence of the semantic representation. Figure 1. System Architecture The logic system then uses a classifier to evaluate the semantic representation against a classification hierarchy of valid logic definitions of full geometry theorems, as well as of many incomplete ways to state them. The results of the classification are passed back to the tutor by the interface module. Based on that, the tutor generates an appropriate feedback to the student’s input [8]. 2.1 Linguistic Inference The linguistic inference module creates the interface between the feature structure unifier and the description logics module. In the process, it also performs two additional inference processes that rely on a combination of linguistic context and domain knowledge: reference resolution and metonymy resolution. The interaction between syntax and semantics is mediated by semantic restriction statements attached to rules in the unification grammar. These statements ensure that the right semantic representation is built compositionally from representations for right-hand side components, and that a reference to the built representation is kept in the feature TUTOR SYNTACTIC PROCESSING